Apparatus and method for augmented reality presentation

ABSTRACT

An augmented reality (AR) system includes a head-mounted display (HMD) with a holographic display, a device for generating virtual reality (VR) light field data, a device for recording light field data of the environment, and a device for combining the light field data of the environment and the VR light field data to form augmented reality (AR) light field data and controlling the holographic display. They AR system further includes a device for correcting the AR light field data on the basis of ophthalmological data of a user. The complete computer-generated AR presentation makes possible a common and uniform adaptation and correction of the presentation in respect of vision defects of the user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of internationalapplication PCT/EP2016/074346, filed Oct. 11, 2016, which claimspriority to German patent application DE 10 2015 219 859.6, filed onOct. 13, 2015, both of which are hereby incorporated by reference intheir entireties.

TECHNICAL FIELD

The disclosure relates to an augmented reality (AR) system, having ahead-mounted display (HMD) with a holographic display and a device forgenerating virtual reality (VR) light field data. The subject matter ofthe disclosure relates furthermore to a method for presenting ARpresentations in or using a system according to the disclosure.

BACKGROUND

VR systems are known from prior public use. They permit presentation ofa computer-generated virtual presentation or reality to a user.

Likewise known are AR systems, in which the user can directly opticallyperceive the environment, and, in addition, VR data appear in his fieldof view. With these AR systems, additional information in the form of VRdata can be made available to a user upon looking at the environment.One known AR system is Google Glass®.

WO 2015/032828 A1 describes an AR system, which provides a correction ofthe introduced VR data on the basis of ophthalmological parameters ofthe user.

SUMMARY

It is an object of the present disclosure to provide an AR system and amethod of presenting AR, which offers good AR presentation and a simpleoption for adapting to different users.

This object is achieved according to the disclosure by way of the systemincluding a device for recording light field data of the environment anda device for combining the light field data of the environment and theVR light field data to form AR light field data and controlling theholographic display, wherein a device for correcting the AR light fielddata on the basis of ophthalmological data of the user is provided.

Firstly, some terms used in the context of the disclosure are explained.

AR systems permit the simultaneous visual perception of both theenvironment (either directly or indirectly by presenting a camerarecording of the environment) and, additionally, of optically superposedor overlaid graphic data for a user, the data being, for example,images, alphanumeric characters, or the like.

HMDs are visual output devices that are supported on the head of a userand can present optical data and images to the user. For example, HMDscan take the form of spectacles, masks, helmets, contact lenses, or thelike. In the context of the disclosure, closed HMDs are typically used,which let the user perceive only computer-generated opticalpresentations and do not permit direct visual perception of theenvironment.

A holographic display is a device that can present a computer-generatedholographic interference pattern image that corresponds in theperception to a stereoscopic image. Holographic displays are known inthe related art and described, for example, in US 2014/0293386 A1.

A device for generating VR light field data produces VR light field datathat permit 3D presentation of the (synthetic) graphic data with aholographic display.

Light field data are described by a vector function that describes theradiance of the light for each point in space and for each direction inspace.

A device for recording light field data of the environment permitscapturing the light field of an observed environment or scene. Thisdevice is typically embodied according to an aspect of the disclosure asa light field camera or plenoptic camera. In principle, light field dataof the environment can also be recorded in a different manner, forexample by way of multiple exposure of a regular image sensor withvarying focal length. The device typically permits the recording andcreation of light field data of the environment in real time.

An exemplary embodiment of the disclosure is a device for combining thelight field data of the environment and the VR light field data to formAR light field data. According to an aspect of the disclosure, provisionis thus made for the AR presentation not to be a mixed presentation of adirect optical perception of the environment with added VR data, but forthe entire AR presentation including the presentation of the environmentto be generated by a computer using the holographic display.

The VR light field data that are combined with light field data of theenvironment can include computer-generated or simulated data, and theycan also include data from a light field camera that is arranged at aspatial distance, for example for telepresence applications.

This procedure in accordance with an aspect of the disclosure permitsthe application of all possibilities for presentation and possiblycorrection of light field data which are presented using a holographicdisplay not only to the VR elements of a mixed AR presentation withdirect perception of the environment, but also to the overallpresentation including the environment and thus the perception of theenvironment.

The system includes a device for correcting the AR light field data onthe basis of ophthalmological data of the user.

The term ophthalmological data of the user refers to data that compriseinformation relating to the individual eyesight or the visioncharacteristics of a user. They can typically be prescription data thatrecord defined types of defective vision of the user. Such prescriptiondata are, for example, sphere; cylinder, and axis (correction ofastigmatism); prismatic correction; higher-order aberrations; addition(correction of presbyopia); and the like.

However, the information relating to the individual eyesight or thevision characteristics can also be other ophthalmological data, forexample prescribed data in connection with the ascertainment ofophthalmological data of a user.

The data can furthermore be sensitivity data that describe for examplethe spectral sensitivity or brightness sensitivity of the eyes. Theknowledge of the spectral sensitivity allows according to an aspect ofthe disclosure for example the correction of color vision defects. Theknowledge of the brightness sensitivity, for example, permits theadaptation of the contrast of the generated AR image to the propertiesof the eyes.

The data can furthermore be biometric data of the user, including forexample pupil distance, axial length of the eyes, orientation of thecenter of rotation of the eyes, and the like.

The disclosure makes it possible to offer an AR presentation to a user,in which both the presentation of the environment and also theadditional visual VR data are corrected together for example forcompensating a defective vision of the user. According to an aspect ofthe disclosure, it is no longer necessary to provide the doublecorrection that is common in AR systems of the related art, in which thedirect visual perception of the environment must be corrected by way oftypical optical elements, such as for example spectacle lenses, which inthe related art, for example, can be placed into the HMD that is used.

The device for recording light field data of the environment istypically in the form of a light field camera that permits recording ofthe light field data in real time. More typically, the device forrecording light field data is coordinated with the viewing direction ofthe user. In this way, the environment is realistically reproduced whiletaking into account the viewing direction. For this purpose, the lightfield camera can be mechanically coupled to the HMD, in particular, beattached thereto. Thereby the light field camera follows the headmovements of the user. Alternatively or additionally, a viewingdirection sensor can be provided in the HMD, which captures eyemovements and the viewing direction and typically controls a trackingmovement of the camera.

Within the context of the disclosure, it is likewise possible for thecamera to be arranged at a spatial distance from the HMD or from theuser of the HMD. Such a spatially remote arrangement is typically usedfor telepresence applications. More typically, such a spatially remotecamera is controllable by the HMD in wire-bound or typically wirelessfashion such that the recording direction of the camera is linked to theviewing direction of the user of the HMD or follows the viewingdirection. By way of example, the HMD can include accelerometers thatcapture head movements of the user and thus provide a control variablefor the tracking movement of the remote camera.

According to an aspect of the disclosure, it is typical for the devicefor generating VR light field data to be connectable to an external datasource. This can be an external data source (for example databases orgenerally the Internet) that contributes content for the VRpresentation.

Alternatively or additionally, provision may be made not only for aconnection to an external data source, but also for a connection to anexternal computer, whose computational power is used for example forprocessing light field data, controlling the holographic display or thelike. This variant of the disclosure allows computationally complexprocessing of light field data to be relocated to high-capacity externalcomputers.

The connection of an external data source or external computer and thesystem according to an aspect of the disclosure or a HMD is typicallywireless, and common electromagnetic or optic wireless connectiontechniques can be used. If available in the relevant environment,optical connections have the advantage that broadband transfer ispossible without difficulty and exposure of the head region of the userto electromagnetic radiation can be avoided.

According to an aspect of the disclosure, provision may be made for theexternal data source to additionally include ophthalmological data ofthe user. This can be beneficial in particular if a system according tothe aspect of the disclosure is used by different users having differentdefective vision. By access to the external data source, automaticadaptation to the respective user can take place. Alternatively, thecorresponding ophthalmological data can be stored locally in the systemaccording to another aspect of the disclosure or in the HMD.

Very common types of defective vision are nearsightedness orfarsightedness, which require spherical correction of the focal plane.Provision may be made according to an aspect of the disclosure for thedevice for combining the light field data of the environment and the VRlight field data to form AR (augmented reality) light field data and forcontrolling the holographic display to be configured for presenting,from the AR light field data, the focal plane that corresponds to thespherical vision defect of the eye for each eye of the user. In thisexemplary embodiment of the disclosure, it is possible in each case togenerate and use only the focal planes that correspond to the visiondefects of the left or right eye of the wearer. Such a procedurerequires comparatively low computational power and consequentlyfacilitates the performance of the calculations for correcting the ARlight field data on the basis of ophthalmological data of the userlocally in the region of the system or HMD.

According to another aspect of the disclosure, it is typical for the HMDor the holographic display thereof to include a spatial light modulator(SLM). Suitable SLMs are familiar to those skilled in the art and do notrequire a more detailed explanation here. A disclosure of suitable SLMscan be found, for example, in WO 2015/032828 A1, which was mentioned inthe introductory part.

The SLM can be embodied according to an aspect of the disclosure forexample as a reflective or transmissive SLM. One example of atransmissive SLM are stacked liquid crystal (LC) displays, as aredescribed, for example, in the publication Huang et. al., “The LightField Stereoscope,” retrievable atwww.computationalimaging.org/publications/the-light-field-stereoscope/.Reference is made to this disclosure.

The system according to an aspect of the disclosure typicallyadditionally has a viewing direction sensor. Capturing the viewingdirection can be useful or required in particular if defective visionthat is dependent on the viewing direction, such as astigmatism, isintended to be corrected in accordance with an aspect of the disclosure.According to another aspect of the disclosure, provision may thus bemade for the viewing direction of the user, which is captured by theviewing direction sensor, to be included in the correction of the ARlight field data on the basis of ophthalmological data of the user.

A further aspect of the disclosure is a method for generating orpresenting AR presentations using a system, or in a system, having thesteps of recording light field data of the environment, generating VRlight field data, combining the light field data of the environment andthe VR light field data to form AR light field data, modifying the ARlight field data on the basis of ophthalmological data of the user, andpresenting the modified AR presentation in the HMD.

As has already been described in the context of the system according toan aspect of the disclosure, it is typical for the modification tocomprise the generation of the focal planes that correspond to thespherical vision defects of the respective eye of the user.

It is furthermore typical for the modification to be affected whiletaking into account the viewing direction of the user that is capturedby the viewing direction sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the disclosure are described below withreference to the drawings, wherein:

FIG. 1 schematically shows an exploded view of a HMD of a systemaccording to an exemplary embodiment of the disclosure; and

FIG. 2 schematically shows a system according to an exemplary embodimentof the disclosure having a remotely arranged camera.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

FIG. 1 shows housing 1, in which two LC displays 2, 3 are arranged at adistance from one another that is determined by the spacer 4. These twodisplays together form an SLM, as described above.

The displays are backlit by a backlight device 5.

The components mentioned are arranged in the closed housing 1, whichhas, at the front side, an opening or cutout 6 that is approximatelyadapted to the head shape of a user and serves for substantiallyenclosing a HMD that is positioned in front of the eyes of a user suchthat it is light-tight.

The look the user casts onto the image that is presented by the LCdisplays is conveyed schematically by way of lens elements indicatedschematically with 7.

Depending on the exemplary embodiment of the display, additional lenselements are not absolutely necessary, but may be desirable, e.g., asUV, blue light, or contrast filters.

A physical separation of the presentation for the right and the left eyemay be useful depending on the display used, albeit not absolutelynecessary in certain circumstances.

The HMD in an exemplary embodiment of the disclosure can have a lightfield camera that is fixedly connected thereto (not illustrated in thedrawing), which can be arranged for example at the side of the HMD thatis remote from the lens elements 7 and automatically follows the viewingdirection of the user as a result of the fixed connection to the HMD.

In FIG. 2, another exemplary embodiment of the disclosure is shown, inwhich the camera 9 can be arranged to be physically separate. In thatcase, devices are typically provided that ensure, by way of a data link10, 11, that the camera, which is arranged at a spatial distance, movesaccording to the head movements of the user of the HMD and consequentlyfollows his or her (imaginary) viewing direction. A camera of this type,which is arranged at a spatial distance, can be used, for example, formaking a system according to the disclosure utilizable for telepresenceevents.

In the exemplary embodiment of FIG. 2, a device for generating VR lightfield data is schematically illustrated as an external computer 8 thatis connected to the HMD via a data link 10. This computer 8 may or maynot be structurally connected to or integrated in the HMD 1.

The foregoing description of the exemplary embodiments of the disclosureillustrates and describes the present invention. Additionally, thedisclosure shows and describes only the exemplary embodiments but, asmentioned above, it is to be understood that the disclosure is capableof use in various other combinations, modifications, and environmentsand is capable of changes or modifications within the scope of theconcept as expressed herein, commensurate with the above teachingsand/or the skill or knowledge of the relevant art.

The term “comprising” (and its grammatical variations) as used herein isused in the inclusive sense of “having” or “including” and not in theexclusive sense of “consisting only of” The terms “a” and “the” as usedherein are understood to encompass the plural as well as the singular.

All publications, patents and patent applications cited in thisspecification are herein incorporated by reference, and for any and allpurposes, as if each individual publication, patent or patentapplication were specifically and individually indicated to beincorporated by reference. In the case of inconsistencies, the presentdisclosure will prevail.

The invention claimed is:
 1. An augmented reality (AR) system forgenerating an AR presentation, the system comprising: a head mounteddisplay (HMD) having a holographic display; a device configured togenerate virtual reality (VR) light field data; a recording deviceconfigured to record light field data of the environment; a controldevice configured to combine the light field data of the environment andthe VR light field data to form AR light field data, control theholographic display, and present, from the AR light field data,generated focal planes that correspond to a spherical vision defect ofan eye for each eye of the user; and a correction device configured tocorrect the AR light field data based on ophthalmological data of auser, wherein the AR presentation presented on the holographic displayincludes the AR light field data formed by the control device, andwherein the light field data of the environment is entirelycomputer-generated.
 2. The system as claimed in claim 1, wherein theophthalmological data of the user are selected from a group consistingof prescription data, biometric data, and sensitivity data.
 3. Thesystem as claimed in claim 1, wherein the recording device configured torecord the light field data of the environment is a light field camera.4. The system as claimed in claim 1, wherein the recording deviceconfigured to record the light field data is coordinated with a viewingdirection of the user.
 5. The system as claimed in claim 1, wherein thedevice configured to generate the VR light field data is connectable toan external data source.
 6. The system as claimed in claim 5, whereinthe external data source further comprises the ophthalmological data ofthe user.
 7. The system as claimed in claim 1, wherein the HMD comprisesa spatial light modulator (SLM).
 8. The system as claimed in claim 1,further comprising a viewing direction sensor.
 9. The system as claimedin claim 8, wherein a viewing direction of the user, which is capturedby the viewing direction sensor, is included in the correction of the ARlight field data based on the ophthalmological data of the user.
 10. Amethod for presenting AR presentations in the system as claimed in claim1, the method comprising: recording the light field data of theenvironment; generating the VR light field data; combining the recordedlight field data of the environment and the generated VR light fielddata to form the AR light field data; modifying the AR light field databased on the ophthalmological data of the user by generating the focalplanes that correspond to the spherical vision defects of respectiveeyes of the user; and presenting the modified AR light field data on theholographic display of the HMD, the modified AR light field dataincluding the recorded light field data of the environment, the recordedlight field data of the environment being entirely computer-generated.11. The method as claimed in claim 10, further comprising capturing aviewing direction with a viewing direction sensor, wherein the modifyingis affected by taking into account the viewing direction of the usercaptured by the viewing direction sensor.